APPGROUP1 Recovery (APP11 processes)

 

                APPGROUP1 (APP11) Purpose:

APP11 processes receive MESSAGEs and MESSAGE related information from MESSAGE sending SITE1s and send to APP02.

They then receive related responses from the APP02 and send these back to MESSAGE senders.

 

Use the following hyperlinks to jump to the desired section of APP11 documentation:

APP11_Recovery_Considerations

APP11_NTM_Control_Commands

APP11_Troubleshooting_Table

APP11_Monitoring_Considerations

 

APP11 Recovery Considerations:

                                Stopping/Restart Processes:

-          Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

 

-          When moving between nodes:

o   ALTERNATE NODES are only to be used when a given SITE1 is having issues and cannot connect to their PRIMARY NODE.

o   DR NODES are to be used when  is having issues and must move the affected SITE1s to another node.

DR NODES must be allocated un-natted nodes. 

No node can support more than one natted address at the same time.

 

-          When stopping/restarting APP11 processes:

1)      Notify the associated SITE1 and work in cooperation with them, as appropriate to situations.

2)      Notify Tech Services if moving APP11 processes to DR nodes and NAT addresses need to change to accommodate move.

3)      Stop the APP11 process.

4)      If NOT moving APP11 to new node, skip to step 6.

a)       If moving the APP11 to a new node, copy the day’s APP11 FIX engine files to alternate node:

a)       Copy :\\data\{APP11}\*.conf, *.in, *.ndx.in, *.out, *.ndx.out files for each APP11 process moving to the alternate node.

b)      If the target folder does not exist on the new node, you must first either create the folder, or copy the entire folder.

c)       If these files are not moved before the process restarts on the new node, there will be a chance of sequence number miscommunications between the MESSAGE sending SITE1 involved and .

d)      If moving a CMA associated APP11 (or FSD) process, also see CMA recovery considerations documentation.

5)      Start the APP11 process.

6)      Open the channel for the process affected and conSITE1 MESSAGE sending SITE1 connects as expected.

 

APP11 NTM Control Commands:

Open OSF Channel:

-          Use NTM Control Utility – Service Control - APP11 – Open OSF Channel to make OSF connection to APP11 possible.

 

Close OSF Channel:

-          Use NTM Control Utility – Service Control - APP11 – Close OSF Channel to make OSF connection to APP11 impossible.

 

Set Inbound Sequence Number:

-          Use NTM Control Utility – Service Control - APP11 – Set Inbound Sequence Number to set  Inbound Sequence Number.

 

Set Outbound Sequence Number:

-          Use NTM Control Utility – Service Control - APP11 – Set Outbound Sequence Number to set  Outbound Sequence Number.

 

Enable 29West Stats:

-          Use NTM Control Utility – Service Control - APP11 – Enable 29West Stats to start collection and display of LBM related stats.

 

Disable 29West Stats:

-          Use NTM Control Utility – Service Control - APP11 – Disable 29West Stats to stop collection and display of LBM related stats.

 

 

APP11 Troubleshooting Table:

APP11 Symptom

Impacts

Response

SITE1 disconnects or Logs out of session

 

Evidenced by:

-          EMT message saying {SITE1} is disconnected and/or {SITE1} is logged out.

 

-          Stats monitor shows disconnected in status column.

 

{SITE1} will be further identified in EMT message by including “LocalFixId” and “RemoteFixID” as configured in APP11Services.xml file within the disconnect message.

 

SITE1 is no longer able to send or receive MESSAGE or MESSAGE related messages with  Matching Engine.

1)      Contact SITE1.

2)      Work with SITE1 and/or Technical Services as necessary to isolate cause of issues and resolve them.

3)      Stop/Restarts of affected application service may help resolve the issue.

 

SITE1 wants to force all MESSAGEs from service to be canceled.

 

Evidenced by:

-          SITE1 calling and requesting all MESSAGEs be canceled.

 

The affected SITE1 wants their risk mitigated by not leaving any open MESSAGEs in the matching engine.

1)      Stopping the SITE1’s APP11 service(s) will also force a “Send AllOrdCxlReq” message to all APP02, canceling all of the affected APP11’s open MESSAGEs.

2)      Use the NTM Control Utility ME option to “Forcibly Cancel MESSAGEs by SITE1” if the SITE1 wants all MESSAGEs canceled, regardless of which APP11 service it may have been sent through.

 

APP11 Monitoring Considerations:

Stats Monitors:
APP11 App Connect Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from MESSAGE sending SITE1s directly to  APP02.
(Direct access Protocol Buffer)

Monitor shows connection status between  fix engine and SITE1, or THIRD PARTY connection status and matching engine, by topic.

PROD MENU:
SITE1switch Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- Status,
- Write Queue

Data is RED.

 

Process is either down or multicast data is not being received by monitor.

1) Check status of process
2) If process is up, call Technical Services.

Status is Disconnected or Open

 

SITE1 is not connected.

1) Use NTM Control Utility APP11 Service Controls to Open Channels.
2) Call Production Control if needed.

Status is Inactive

 

THIRD PARTY communications has been disabled between the  fix engine and the matching engine.

1) Check status of process
2) If process is up, call Production Support.

Write Queue is non-zero values and not reducing as expected.

SITE1 may not be processing as expected.
1) Call SITE1 and work with Technical Services if necessary.

 

 

Stats Monitors:
APP11 App Processing Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from MESSAGE sending SITE1s directly to  APP02.
(Direct access Protocol Buffer)

Monitor shows processing statistics of application.

PROD MENU:
SITE1switch Monitoring Menu

To Exit:
Close Window

- Color of data in columns
- InCount,
- OutCount,
- LastInTime,
- LastOutTime

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

No data is displayed.

No data has been generated by MESSAGE sending SITE1. 
1) Check APP11 fix message files. If all sizes are zero, no traffic has been generated.
2) Call SITE1 if necessary.

InCount is not less than, or equal to OutCount.

SITE1 may not be receiving all matching engine responses expected.
1) Check APP11 FIX message files to conSITE1 inbound messages and outbound messages.
2) Call SITE1 if necessary.

 

 

Stats Monitors:
29West APP11 LBM Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from MESSAGE sending SITE1s directly to  APP02.

Monitor shows THIRD PARTY statistics (by topic) for APP11 processes, receiving from Matching Engine processes, and sending to Matching Engine Processes.

PROD MENU:
THIRD PARTY LBM Monitor Menu

To Exit:
Close Window

- Color of data in columns
- ContextName,
- Service,
- Topic_name,
- Type,
- Rate,
- Persistence,
- MsgCount

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Statistics are not shown for APP11 process as expected.

Process may not be up, or THIRD PARTY Stats have not yet been enabled for process.
1) Check status of process
2) Via NTM Control APP11 Service Controls, enable THIRD PARTY stats for process.

Rate and/or MsgCount values are not incrementing as expected.

SITE1s may not be receiving messages as expected.
1) Check LBM APP11 files to conSITE1 THIRD PARTY related processing.
2) Call Production Support if necessary.

 

 

 

 

Stats Monitors:
29West APP11 LBM RCVRM Stats

To Start:

Key Indicators to Monitor:

Symptom:

Response:

Processes facilitate communications from MESSAGE sending SITE1s directly to  APP02.

Monitor shows THIRD PARTY "receiving" statistics (by sender, and topic) for APP11 processes, receiving from Matching Engine processes.

PROD MENU:
THIRD PARTY LBM Monitor Menu

To Exit:
Close Window

- Color of data in columns
- ContextName,
- Service,
- Topic_name,
- Lost-Recovered,
- Lost-Unrecovered-Txm,
- Lost-unrecovered-tmo,
-Msgs_rcved

Data is RED.

Process is either down or multicast data is not being received by monitor.
1) Check status of process
2) If process is up, call Technical Services.

Statistics are not shown for APP11 process as expected.

Process may not be up, or THIRD PARTY Stats have not yet been enabled for process.
1) Check status of process
2) Via NTM Control APP11 Service Controls, enable THIRD PARTY stats for process.

Msgs_rcved values are not incrementing as expected.

SITE1s may not be receiving messages as expected.
1) Check LBM APP11 files to conSITE1 THIRD PARTY related processing.
2) Call Production Support if necessary.

Lost-unrecovered values are non-zero.

NOTE: Lost-recovered values may also indicate problems, but that 29west auto-recovered lost messages.

SITE1s may not be receiving messages as expected.
1) Check LBM APP11 files to conSITE1 THIRD PARTY related processing.
2) Call Production Support if necessary.